ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

Note that fully supporting Parquet encryption requires being able to get write and read properties per-file, which is why the existing ability to set WriterProperties isn't sufficient, and why WriterPropertiesFactory::create_writer_properties is called per file and requires a file path. This allows generating new random data encryption keys per file and performing tasks such as specifying a per-file AAD prefix or supporting the external storage of encryption keys that can be looked up using the file path.

@rtyler @roeap @alamb Tagging you here per our previous discussion on adding encryption support to delta-rs.

I have marked this pull request as draft. This does not compile as is, I can come back to it once it is able to compile and pass unit tests

I have marked this pull request as draft. This does not compile as is, I can come back to it once it is able to compile and pass unit tests

@rtyler OK. It seems that when I auto-merged the main branch it introduced a build error. I have resolved this and the code is once again building and passing unit tests.

@corwinjoy - awesome to see this come to fruition! Will find some time to give this a review hopefully tomorrow.

At first glance one quick question. Do we see a way to "bundle" the datafusion specific stuff a bit more? It's a bit hard to keep track of all the individual flags while reviewing :)

@roeap

At first glance one quick question. Do we see a way to "bundle" the datafusion specific stuff a bit more? It's a bit hard to keep track of all the individual flags while reviewing :)

What we did to minimize this dependency is define an abstract FileFormatOptions trait. Everything just passes around a FileFormatRef defined as Arc<dyn FileFormatOptions>. Then, only when needed, do we grab final table options or writer properties. Furthermore, we've gated these instances of getting final details behind three function calls in file_format_options.rs:

pub fn build_writer_properties_factory_ffo(
    file_format_options: Option<FileFormatRef>,
) -> Option<Arc<dyn WriterPropertiesFactory>> {...}

pub fn to_table_parquet_options_from_ffo(
    file_format_options: Option<&FileFormatRef>,
) -> Option<TableParquetOptions> {...}

pub fn state_with_file_format_options(
    state: SessionState,
    file_format_options: Option<&FileFormatRef>,
) -> DeltaResult<SessionState> {...}

There might be some ways to refine this further, but in general we've tried to isolate and abstract these file properties where possible and not require datafusion.

@roeap From a user point of view, we've tried hard to make the settings as easy as possible. This can be seen in crates/deltalake/examples/basic_operations_encryption.rs. Here, we demonstrate different kinds of operations on tables. (We have a more formal unit test at crates/core/tests/commands_with_encryption.rs). Thes code examples all look like ordinary operations; all we needed was a common function call when creating DeltaOps:

async fn ops_with_crypto(
    uri: &str,
    file_format_options: &FileFormatRef,
) -> Result<DeltaOps, DeltaTableError> {
    let prefix_uri = format!("file://{}", uri);
    let url = Url::parse(&*prefix_uri).unwrap();
    let ops = DeltaOps::try_from_uri(url).await?;
    Ok(ops.with_file_format_options(file_format_options.clone()))
}

Calling with_file_format_options is sufficient to apply the needed encryption settings for all operations.

@ion-elgreco OK. I have applied the changes you suggested from two weeks ago and re-merged the latest from main. So, I am ready for another review when you get the chance. Thanks again for the suggestions!

@ion-elgreco OK. I have added changes to address your latest requests.
In addition, I have merged the latest main.
Plus, I have added the update from @jgiannuzzi to support async KMS for the tests and example.
Please let me know if you have any further questions or are ready for another review.

@roeap @roeap for me it looks in a mergable state, maybe you guys can give a final thought on it?

Perhaps we should put this behind an encryption feature gate

I had thought about doing this with an encryption feature gate, but left it out for this PR for a few reasons:

1. Adding the gate makes both the code and usage more complex. Making it more of a hassle both to use and debug this feature.
2. Relative to the size of the delta-rs project, disabling the encryption feature won't save much on size.
3. This may be cleaner to add as a follow-up rather than mixed in with an already longish PR.